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ABSTRACT 

A team approach to the use of qualitative methods in 
a study of high quality out-of-class experiences for undergraduate 
students at 14 institutions of higher education is described. Four 
research questions examined: the physical and organizational features 
characterizing the institutions; the nature, of institutional policies 
related to out-of--?' ass activities; opportunities available for 
student involvement; and how elements of institutional culture and 
subcultures influence student involvement. Issues of qualitative 
research addressed include team composition, data collection, 
observations, data analysis, establishing trustworthiness and 
credibility, transferability of findings, and dependability and 
confirmability. Among conclusions were that multisite qualitative 
research using multiple investigators is time consuming and 
expensive, and that the composition of the research team is critical* 
Such research does work, however, and in this case provided rich and 
accurate descriptions of individual institutions and factors and 
conditions across institutions that seem to contribute to 
high-quality out-of-class experiences. Contains 34 references. 
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INTRODUCTION 

Qualitative data tend to overload 
the researcher badly at almost 
every point; the sheer range of 
phenomena to be observed, the 
recorded volume of notes, the time 
required for write-up, coding, and 
analysis can all be overwhelming 
(Miles, 1979, p. 590). 

This caveat from Miles (1979) notwithstanding, qualitative 
research methods are receiving increasing attention as being 
particuliirly useful for exploring and understanding some of the 
"hard-to-measure" (Crowson, 1987, p. i) characteristics of 
institutions of higher education (e.g., cultures, values, norms, 
beliefs) (Crowson, 1987; Peterson, 1985). At the same time, 
relatively few qualitative studies of colleges or universities 
have been conducted (Crowson, 1987; Tiemey, 1985). Examples of 
qualitative investigations involving multiple sites and multiple 
investigators are expecially rare, despite their potential for 
providing in-depth understanding, as well as broad comparisons, 
of institutional contexts (Crowson, 1987; Herriott & Firestone, 
1983) . 

The purposes of this paper are (a) to describe a team 
approach to the use of qualitative methods in a study of high 
quality out-of -class experiences for students at 14 institutions 
of higher education, and (b) to add to knowledge about the 
processes, pleasures, and pitfalls of the use of such an approach 
in higher education research. First, we provide a brief overview 
of the purposes of the study. Second, the theoretical framework 
used in the study is identified, followed by a description of 
research methods, including data sources, data collection, and 



data analysis. Finally, we discuss our findings and conclusions 
with regard to the research process. 

OVERVIEW OF THE STUDY 
Although considerable attention has been given to assessing 
the academic outcomes of college (Adelman, 1987; Ewell, 1985), 
few studies have focused on student life outside the classroom. 
If 48 hours of a typical college student's week are devoted to 
attending class and studying (Boyer, 1987), about two-thirds of 
the time in a given week is spent on other activities. 
Nevertheless, out-of-class experiences are taken for granted or 
lightly regarded as a positive educational force on many 
campuses. However, students who actively participate in both 
academic and out-of-class activities become better integrated in 
the academic and social life of the institution (Chapman & 
Pascarella, 1983; Tinto, 1986) and tend to be happier, are more 
likely to persist to graduation, and exhibit higher levels of 
achievement and personal development than students who are not 
involved (Astin, 1977; Pascarella, 1980; Tinto, 1987). Boyer 
(19.37) concluded that "the effectiveness of the undergraduate 
experience relates to the quality of campus life and is directly 
related to the time students spend on campus and the quality of 
their involvement in activities" (p. 180) . 

For our purposes, a high quality out-of-class experience was 
defined as students' active participation in out-of-class 
activities which complement the academic purposes of the 
institution. Astin (1977, 1985), Boyer (1987), and others (e.g.. 



NASPA, 1987) have identified some elements of out-of-class 
experiences that support the academic program (e.g., residential 
honors programs, cooperativva education-work programs) . This 
study was designed to extend the work of Astin, Boyer ar?/» the 
Carnegie Foundation, and Pace (1980, 1982, 1984) by looking at a 
group of institutions reputed to provide high quality out-of- 
class experiences for students, and takes into account 
institutional history, campus traditions, student participation 
in out-of-class experiences, and other factors. More 
specifically, the following research questions guided the study: 
(a) What physical and organizational features characterize these 
institutions?; (b) What is the nature of institutional policies 
related to out-of-class activities?; (c) What opportunities are 
available for student involvement?; and (d) How do elements of 
institutional culture and subcultures (e.g., assumptions, values, 
traditions, language) influence student involvement? 

THEORETICAL FRAMEWORK 
The inquiry methods used in this study were qualitative, 
producing data in the form of words and analyzing data by lueans 
of the use of human instruments (i.e., the inquirers) (Lincoln & 
Guba, 1985). The recognition on the part of many researchers 
that organizations are complex entities with characteristics 
(e.g., loose coupling (Weick, 1980), organizational cultures 
(Dill, 1932; Kuh & Whitt, 198S; Morgan, 1986; schein, 1985) that 
are difficult to measure or quantify has encouraged an interest 
in qualitative research methods (Crowson, 1987) . Qualitative 



methods are considered to be superior to other research methods 
for identifying values, assumptions, expectations, and behavior 
(Goetz & Lecompte, 19C4; Van Maanen, 1979a), sueh as those of 
faculty, students, and administrators that may influei^c-e- the out- 
of-class experiences of students. 

Fundamental principles of qualitative research include: (a) 
a search for understanding, (b) investigator proximity, (c) 
inductive analysis, and (d) an appreciation of the value-laden 
nature of inquiry (Crowson, 1987). A description of each of 
these principles follows. 

The "central goal" (Crowson, 1987, p. 4) of qualitative 
research is understanding, rather than identification of causes 
or generalizability. That is, the investigator seeks to look at 
and appreciate the setting studied from the perspectives of the 
persons within it and strives to interpret what is seen in light 
of those persons' frames of reference. in addition, the 
investigator's findings and interpretations must be presented in 
such a way that both "insiders" (i.e., participants of the 
setting) and "outsiders" achieve greater understanding of the 
setting and its cultures (Schein, 1985) . 

In order to attain undarstanding, the investigator must 
study behaviors as they occur and hear the thoughts and words of 
participants first-hand (Crowson, 1987; Van Maanen, 1979a). 
"Immersion" (Crowson, 1987, p. 7) in the setting enables the 
researcher to gather multiple perspectives and obser-ze many and 
varied behaviors and events, in this way she or he can provide 



VOie thick description needed for outniders (and insiders) to 
understand the setting. 

Analysis of qualitative data is intended to build 
understanding inductively— that is, building w^.derstanding from 
observed data— rather than deductively (i.e., formulating 
hypotheses and identifying data categories or variables a 
priori) (crowson, 1987; Lincoln & Guba, 1985). Because at the 
outset of the investigation the researcher is not aware of all of 
what he or she does not Jcnow (Lincoln & Guba, 1985), the 
development of preconceived hypotheses or explanations of what is 
going on is likely to inhibit consideration of all possible 
meaningful behavior and events. An inductive approach allows the 
setting and the researcher's growing knowledge of the setting to 
direct the research process. 

There has been some debate about whether the researcher 
should have done any prior theorizing (e.g., using predetermined 
conceptual or theoretical frameworks) before entering the setting 
to be studied (Crowson, 1987; Lincoln & Guba, 1985; Miles & 
Huberman, 1934) . some (cf , Lincoln & Guba, 1985) assert that the 
use of such frameworks is as inhibiting to "noticing" as 
hypotheses, m the study descrikied here, we began data 
collection with a conceptual framework in mind, recognizing that 
"the investigator's theoretical point of view does color a study 
and will shape the inductive vision of reality that appears in 
the researcher's findings" (Crowson, 1987, p. 9). 

In a similar vein, principles of qualitative research 
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include recognition that inquiry is vilue-laden, rather than 
value-free. The researcher is bound by the values of his or her 
inquiry paradigm (e.g., positivist or naturalist) and cultural 
context, as well as the values extant the research setting; 
these values influence what the researcher sees and the meanings 
he or she makes of what is seen (Crowson.- 1987; Lincoln & Cuba, 
1985; van Maanen, 1979b). The influence of investigator and 
contextual vaflues can be offset to some extent by rigorous and 
prolonged focus on the perspectives of insiders (Crowson, 1987). 
The more time the investigator can spend in the setting and the 
more thorough her or his investigation of "natives" views, the 
more likely it is that those views will be portrayed accurately. 
Steps taken in this study to cope with this and other 
methodogical problems will be described in the next section. 

RESEARCH METHODS 
In this section, we describe the methods used in the study, 
including composing the research team, selecting data sources, 
collecting and analyzing data, and establishing trustworthiness. 

The Research Team 
Effective field research using qualitative methods requires 
that the inquirers be familiar not only with appropriate inquiry 
techniques but also with the phenomena under study. Thus, 
understanding is not only a goal of qualitative research, but "a 
necessary precondition of useful fieldwork" (Crowson, 1987, p. 
13) . Having some familiarity with the setting—or type of 
setting--, people, and processes to be studied helps the 
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investigator to be prepared for at least seme of what she or he 
should look for or might see. Also, a Jcnowledgeable investigator 
is more likely to be credible to the "natives" in the setting 
(Crowson. 1987) and so may htive to spend less time obtaining 
their cooperation and winning their trust. 

At the same time, too much familiarity can be inhibiting; 
unusual events or patterns may be ignored and/or alternative 
explanations or questions may be dismissed. The use of multiple 
investigators ameliorated this concern somewhat because each of 
us had a unique viewpoint and so saw things differently, as well 
as saw different things. The process of incorporating varied 
perspectives into a single set of case reports and conclusions 
will be discussed in the section on data analysis. 

In addition, the ambitious scope of this particular project 
required multiple investigators. one or two individuals could 
not have conducted the number of interviews and observations 
required to provide a rich description of fourteen "involving 
colleges". 

With these qualifications in mind, the research team was 
composed of nine members: four faculty, including a former 
college president' and university provost, a former academic dean 
and department chair, the head of a preparation program in 
college student affairs administration, and a former dean of 
students; three student life administrators, including one chief 
student affairs officer with 25 years of experience at private 
institutions of higher education (iHEs) , one associate vice 



president for student affairs at an urban university who also had 
extensive experience with residence life, and one dean of 
students who served at both commuter and residential 
universities; and two graduate students, one of whom had 
experience in student affairs administration at a womon's college 
and at large public universities. 

Also, the team included three women and one black man. We 
discovered that there were situations (e.g., individual 
interviews with senior students, a site visit to a predominantly 
white university during a time of racial tensions, visits to the 
women's college and the historically black college) where female 
or black investigators were able to establish more effective 
rapport and make the respondents feel more at ease and more open 
to sharing their feelings and attitudes. 

Data Sources 

In May and June, 1988, a modified Delphi technique was used 
to identify a small number of ihEs reputed to provide high 
quality out-of -class experiences for undergraduates. Fi.cty-eight 
experts were identified to represent a variety of viewpoints and 
constituencies in higher education. The experts included higher 
education scholars with long-standing interest in the college 
student experience (e.g., Astin, Pace, Riesman) (n=20) ; directors 
of regionsa accreditation associations (n=C) ; representatives of 
associations or agencies with special interest in higher 
education and the undergraduate experience (e.g., American 
Council of Education, American Association of Higher Education, 



Education Commission of the states, National Association of 
Student Personnel Administrators, campus COMPACT) (n=i4) ; college 
and university presidents, including representatives of single- 
sex, historically black, . and urban institutions (n=7) ; and chief 
student affairs officers, sevaral of whom were current or former 
presidents of the American College Personnel Association or the 
National Association of Student Personnel Administrators Cn=ii) . 
Of the 58 experts originally identified, eleven were unable to 
participate (e.g., due to illness, due to a stated lack of 
knowledge about out-of-class experiences); therefore, 47 of the 
expert panel members participated in one or both rounds of the 
nomination process. 

The experts were asked to identify ihEs ncted for the high 
quality out-of-class experiences they provided for 
undergraduates. They were to choose up to five institutions in 
the following categories: (a) residential colleges with fewer 
than 5000 students; (b) residential colleges/universities with 
5000 o.. more students; (c) urban-commuter institutions— i.e, , 
those with a high proportion of com-uting and part-time students; 
(d) single-sex colleges; and (e) historically bi.ack colleges. 
These categories were selected as representative of the divarsity 
of four-year institutions in the .American higher education 
system. 

In the first round, 252 institutions were nominated: 78 
small residential institutions, 67 large residential 
institutions, 49 urban-commuter institutions, 33 single-sex 



colleges (some of which were removed because they were no longer 
single sex— e.g., Haverf ord) , and 25 historically black colleges. 
A list of those institutions receiving two or more nominations 
(n=107s 27 smll residential, 29 large residential, 21 urban- 
ccamuter, 16 single-sex, 14 historically blacJc) was sent to the 
panel of experts. They were asked to identify those that they 
believed provided high quality out-of-class experiences for 
undergraduates. m the second round, 85 institutions were 
nominated by four or more experts: 20 small residential, 23 large 
residential, 16 urban, 15 single seX; and li black. 

After the nominating process was completed, approximately 
one-fourth (n=12) of the experts was intsirviewed by phone about 
the criteria they had used in making nominations. This 
information was used by the research team in making decisions 
about the institutions to be visited. 

The research team met for three days in August 1988 in order 
to select, from among those institutions nominated by the expert 
panel members, the colleges and universities to be included in 
the study. Focusing primarily on the results of the second round 
of nominations and expert interviews, the team engaged in a 
series of discussions to identify institutions to visit. Three 
decision rules emetged during the discussions. First, the team 
decided that, beceiuse some publications had already focused on 
some of the institutions that received many nominations (e.g., 
Haverford—Heath, 1968; Swarthmore— Clark, 1970), the study would 
be more likely to expand knowledge about higher education and the 
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undergraduate experience by including some colleges and 
universities about which less had been written. Second, an 
effort was made to include institutions from different 
geographical regions of the United States; we assumed that 
regional context influences, in some ways, both the student body 
and the irstitution and, hence, the student experience (Kuh & 
Whitt, 1988) . Finally, we attempted to achieve a balance between 
public and private institutions, assuming that form of control 
also affects student experiences (Astin, 1977) . 

Fourteen institutions were selected for inclusion in the 
study: .-Tour small residential colleges (Berea, Earlham, Evergreen 
State, Grinnell), four large residential universities (University 
of California at Davis, lowa State, Miami University of Ohio, 
Stanford) , four urban-commuter institutions (University of 
Alabama-Birmingham, University of Louisville, University of North 
Carolina at Charlotte, Wichita state University), one single-sex 
college (Mount Holyoke) , and a historically black institution 
(Xavier University of New Orleans) . 

No scientific sampling process is claimed. However, through 
the polling and interviewing of experts, and the development and 
review of the final list of nominations by the research team, we 
were satisfied that this set of colleges and universities could 
offer useful information about high quality out-of-class 
experiences for undergraduate students. 

In late August and early September, the chief student 
affairs officer at each institution was contacted to describe the 
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study and the process by which the 14 institutions were selected, 
and to seek permission to include their institution in the study. 
All of the institutions agreed to participate. The oral 
agreement was confirmed with a letter and a request for written 
information about the institution (e.g., institutional histories, 
catalogs, admissions publications, descriptions of student 
characteristics) ; these materials would be reviewed prior to the 
campus visit. Also, a list of persons to be interviewed (e.g., 
president, chief academic officer, chief student affairs officer, 
faculty, students) was provided. The means by which this list 
was generated will be discussed in the section on data 
collection. 

The first roxind of site visits was conducted by teams of two 
to four (depending on the size of the institution) investigators 
from mid-September through early December, 1988; the teams 
typically spent three or four days at each institution, again, 
depending on the size of the institution and scheduling 
constraints. After the first round of visits, the research team 
met to decide if there were institutions that required a second 
visit in order to learn all we thought we needed to know. Second 
visits were made to 12 institutions by teams of two to four 
investigators between January and May, 1989. 

Data Collection 

Data collection and analysis were conducted concurrently. 
In this way, we we::e able to use existing data to inform 
collection and interpretation of additional data (Lincoln & Cuba, 
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1985; Miles & Hxiberman, 1984). For axamplo, the process of 
data analysis, we WGre able t» identify add.-ltional cjuestioncs that, 
needed to be asked or gaps in our understanding of a particular 
institution. Qualitative methods of data collection were used, 
i:icluding interviews, observations, and document analysis. 
P^ spondents 

Because all of the events and people in a setting cannot be 
studied, some sort of samplii.j process must be used, although 
this necessarily limits the scope of the research and provides 
some focusing of the researchers' attention (Crowson, l?'??; Miles 
& Huberman, 1984). Initial selections of what to observe and 
whom to interview were guided by our theoretical framework and 
knowledg(S of IHEs and undergraduate student experiences with the 
recognition that our sources of information would increase and 
change as the study proceeded (Crowson, 1987) . 

The institutional contact person (e.g., chief student 
affairs officer) was asked to schedule the initial round of 
interviews. Our selection of interview respondents was based on 
status sampling (Dobbert, 1984). in this instance, status 
sampling required that interviews be conducted with the 
president, chief academic and student affairs officers and their 
principal assistants, faculty members, professional staff who 
worked directly with students, student leaders, and other 
students. The principle of inclusion was stressed to the contact 
person; that is, wa needed to gather information from as many 
perspectives as possible (Miles & Huberman, 1984; Schein, 1985). 
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For example, v-e wanted be certain that we talked with students 
who held formal leadership roles as well as some who were not 
well-integrated into t;.e student social system. We also wanted 
to talk with faculty who may not have had much contact with 
students outside of class, as well as those who served as 
advisors to student orgaiiizations . 

once at the site, we employed a variant of snowball sampling 
(Crowson, 1987; Dobbert, 1S84) to expand our pool of respondents. 
At the conclusion of each interview, respondents were asked to 
identify others whose opinions and/or out-of-class activities and 
experiences differed froid their own (e.g., students who seemed 
less (or more) involved in campus life, faculty who had been at 
the institution for a longer (or shorter) period of time). We 
also did impromptu inteirviews in cafeterias, library foyers, 
student centers, residence halls, and other living units (e.g., 
fraternity and sorority houses) . 

All respondents were asked to sign a consent form giving 
their permission to use information obtained from them in the 
study, including a commitment to seek additional permission to 
quote them directly. Respondents were assured that their 
participation was voluntary and that they could withdraw from the 
study at any time (Dobbert, 1984), although no one did. 
Interviews 

Interviews were conducted to obtain respondents' 
constructions, as well as to confirm and expand information 
already obtained (Lincoln & Guba, 1985) . Although the degree of 
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structure imposed on the interviews varied from less to more as 
the investigation proceeded (and wc knew more about what we 
needed to know) (Crowson, 1987; Lincoln & Guba, 1985), a set of 
questions was developed for each category of respondent (i.e., 
president, student, etc.). Initial questions were developed from 
the research questions and purposes and the interview protocols 
were used by all investigators at all of the sites. Questions 
were added and deleted by the team as interviewing progressed and 
additional questions were necessary for clarification (e.g., in 
response to contradictory information about faculty involvement 
with students out of class) or to obtain additional information 
(e.g., about differences across institutional types). Thus, the 
respondents generated additional questions for the study. 

Individual interviews and focus groups (Merton, Fisk, & 
Kendall, 1956; McMillin, 1989) were the primary methods of data 
collection. Focus groups are semi-structured discussion groups 
that meet only once and concentrate on a specific topic (e.g., 
factors related to students' out-of -class experiences). Focus 
groups have been found to be particularly useful for obtaining 
in-depth information about attitudes, values, and beliefs which 
may not be apparent in observations of behavior or individual 
interviews, as well as for generating ideas and insights at the 
outset of a study (McMillin, 1989). The interaction and 
discussion among focus group members in response to open-ended 
questions from an interviewer can elicit rich information about 
respondents' experiences and interpretations as respondents' 
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comments and insights build from, and upon, one another 
(McMillin, 1989). 

We found that the focus groups were a very fruitful source 
of information as the ideas or opinions of one individual would 
trigger new ideas or similar or contradictory opinions from 
others. The focus groups turned out to be, for the most pari:, 
lively group discussions rather than interviews. The »ain tasks 
of the investigator were keeping the group on the topic (e.g., by 
asking another question when someone paused to take a breath, or 
by summarizing the sense of the discussion) and making sure that 
everyone had an opportunity to speak, while at the same time 
taking notes and monitoring the tape recorder. Almost every 
group had at least one or two particularly outspoken members and 
at least one or two people who had to be drawn out, a situation 
that requircjd the investigator to perform the role of gatekeeper. 

Interviews were recorded by means o2 a tape recorder so that 
all information could be retrieved. Transcripts were made of 
interviews that were deemed to be especially useful (e.g., 
student leaders, minority students, faculty). Interview data 
were compiled by the investigators on interview summary forms 
(Miles & Huberman, 1984) in order to identify themes, questions, 
and reactions generated by each interview. This information was 
used to develop additional questions and during data analysis. 
Observations 

A secondary source of data was observations of programs, 
events, and activities that took place during the campus visits. 
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Observations were considered to be secondary because they were 
typically used to generate topics for interviews (Barley, 1983). 
Observations fell into three categories? (a) regularly-scheduled 
events (e.g., convocations, concerts), (b) spontaneous events 
(e.g. , frisbee matches) , and (c) events conducted for the purpose 
of our visit (e.g., residence hall tours). As the study 
proceeded, we were able to identify more effectively the events 
and activities that would be fruitful sources of information 
(e.g., student interactions in public gathering places, meetings 
of student organizations) . 

We did not actively participate in the events observed; 
rather, we recorded notes and impressions (Dobbert, 1984) . 
Points for clarification and questions were addressed later to 
appropriate individuals. Data from observations were recorded on 
observation summary forms (Miles & Huberman, 1984) in order to 
facilitate the process of identifying further questions and 
emergent themes. 
Documents 

Documents were another secondary source of information, and, 
like observations, provided topics and questions for interviews. 
Documents were also used to describe and understand the 
institutional context (Dobbert, 1984). Investigators obtained 
documents in advance of the visits. The following documents were 
found to be particularly useful: handbooks (e.g., policy, 
procedure, faculty, student, and staff), promotional pamphlets 
(e.g., admissions viewbooks, student organization recruitment 
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brochures), institutional mission and goal statements, 
institutional histories, and other documents that referred to the 
integration of students* out-of-class experiences with the 
academic mission of the institution. in addition to printed 
documents, we reviewed video tapes and slide presentations used 
for institutional advancement and recruitment purposes. 

Relevant documents also became apparent during the campus 
visits. These included student newspapers, planning documents, 
and "table tents" and posters advertising events and activities. 

Data gathered from document analyses v/ere recorded on 
document summary forms (Miles & Huberman, 1984). Information 
from the forms was used to generate questions for respondents as 
well as to develop constructions of the institutional context for 
the purposes of data analysis. 

The forms used to siommarize interviews, observations, 
document analyses were developed, using Miles and Huberman (1984) 
as a guide, by two of the team members prior to the first two 
campus visits. Each of the three forms was a different color, so 
that they could be easily identified (the study generated a lot 
of paper). The forms were changed slightly (e.g., providing more 
space to deal with questions and responses) in response to 
feedback after these visits. A large stack of blank forms was 
mailed to each investigator. 

One of the team members assigned to a site visit served as 
site visit coordinator. This person was responsible for working 
with the campus contact person to arrange interviews, lodging. 
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and airpoirt transportation (if necessary). The site visit 
coordinator also worked with site team members to decide who 
would do which interviews, and dealt with any scheduling problems 
that arose. There were cases in which delayed flights or campus 
emergencies interfered with the interview schedule; we learned 
early to be flexible (e*g., conduct phone interviews, combine two 
individual interviews) and to assume that things would not always 
go as planned. 

Data Analysip 

The process of analyzing the data collected in this study 
was complicated, not only because of the large amount of data 
obtained by nine investigators aw .ourteen institutions, but also 
because of our need to focus simultaneously on analysis of data 
within the individual institutional sites and across sites, our 
experience reflected Miles' (1979) caution that "when several 
f ieldworkers • efforts must be coordinated . . . much energy is 
required to make data systematically 'comparable'" (p. 590). our 
decision at the outset of the study to use common summary forms 
and data coding facilitated both within-site and cross-site data 
analysis. A description of these processes follows. 
Within-site AnalY 3j,3 

A coding scheme was developed to identify categories for the 
purpose of organizing and retrieving data (Miles & Hubsrman, 
1984) and to mitigate against producing "an incoherent, bulky, 
irrelevant, meaningless set of observations" (Miles, 1979, p. 
591) . Categories encompassed a single theme, containing those 
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units of data that related to the same content (Lincoln & Cuba, 
1985) . A preliminary list of category codes was formulated by 
the research team from the conceptual framework, objectives, and 
questions of the study. These categories vere: (a) the role of 
institutional agents regarding out-of-class experiences, (b) 
description and role of student subcultures, (c) description and 
role of institutional history and traditions, (d) description and 
role of institutional policies and practices, (e) description and 
role of institutional mission, (f) characteristics of student 
involvement in out-of-class life, (g) tentative explanations and 
speculations, and (h) other (creating additional categories as 
necessary) . Each of the categories was discussed by the team so 
that all could understand and agree upon tr \ category definitions 
(Miles & Huberman, 1984) . 

At the end of each day of a campus visit, the members of the 
site team met (usually over dinner) to talk about their 
experiences. These meetings were intended (a) to generate 
questions for future interviews, (b) to identify events, 
activities, or documents that should be observed, or respondents 
who shvwld be interviewed, (c) to begin (or, in the case of later 
first-round or second-round visits, continue) the process of 
identifying themes and patterns in the data collected— that is, 
to begin to analyze the data, and (d) to give tired researchers 
an opportunity to "re-charge" by recounting the highlights and 
unusual experiences of their day. The on-site team meetings were 
recorded, either on tape or in witten notes, for use in writing 
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the case report and planning for the second round of visits. 

After each site visit, each investigator recorded his or her 
interview and field notes on interview summary forms (Miles & 
Hubermai, 1984). The forms provided space to record questions 
and responses; issues and themes that emerged in the interview; 
salient or interesting points; and additional questions to be 
pursued. m addition, each investigator completed a case 
analysis form, in which data from interviews, observations, and 
documents were placed in the categories developed by the group. 

These forms, as well interview tapes and notes, were 
forwarded to the investigator designated as the site coordinator. 
The first task of the site coordinator was to compile all of the 
site data, including notes from team meetings on-site. The 
coordinator then assigned the data to categories; case analysis 
forms from the other investigators were used as a means to assess 
the completeness of these categories. if necessary, additional 
categories were developed in order to include all of the site 
data (Miles & Huberman, 1984). Categorization at this point 
served two purposes. First, having the site data in categories 
enabled the development of a case report that summarized findings 
and conclusions for the first visit. Second, categorization of 
the site data provided a basis for analysis of data across sites, 
a process which is described below. 

The case report of the first visit served as an "interim 
site summary" (Miles & Huberman, 1984, p. 75) , synthesizing what 
was known about the site and identifying remaining questions to 



be explored. The case report was then circulated among all 
members of the research team in order to inform data collection 
at other sites. The report was also sent to respondents at the 
site in order to ;*i.low thert to confirm or deny the investigators' 
constructions of their words and feelings (Lincoln & Cuba, 1985; 
Miles, 1979). "corrected" case reports were returned to the 
project director by the contact person at the site and 
respondents' suggestions were incorporated into revised case 
reports . 

Perhaps as a result of our purpose— to discover what is 
happening at places that are doing something well— we tended to 
focus on the positive: what is special about this institution? 
what have been the highlights of your experience here?, and so 
on. Thus, feed'oack from some respondents was that we were too 
positive and too polite in our first -round reports. The 
debriefing process, as well as second-round interviews, provided 
us with what respondents felt was more a more balanced picture of 
their institutions. 

During the second round of site visits, debriefing (Lincoln 
& Guba, 1985) meetings were held with groups of respondents, many 
of whom had been interviewed in the first round. They were asked 
about their reactions to the case report; Did it fit their view 
of the "reality" of their institution? Were there important 
people and events that had been ignored in the first round? Had 
the investigators misinterpreted or misunderstood something that 
had been communicated? What else did we need to know in order to 
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understand and accurately repiresent the undergradua.te experience 
at their institution? our interactions with 'esponden^.s in tha 
debriefing sessions enabled us to further develop our 
understanding of, the .Institutions from the poirspectives of 
insiders and to alter the case reports as necessary to conform 
more closely to those perspectives. Further discussion of 
respondent debriefings can be found in the section on 
trustworthiness . 

After the second roim^ of visits, tlie process used to record 
and analyise site data during bhe first round was repeated. New 
data were incorporated into tU^ revised case reports and the-^e 
reports were once again sent to respondents at the site for their 
reactions and suggestions. A "final" case report was developed 
for each site after all feedback had been received (there may 
never be such a tiing as a £ir i report; wo still r^-.eive, and 
incorporate, occasional BITNET comments and suggestions from 
persons at the sites) . 
Cross-site Analy sis 

Data from all of the first-round campus visits were compiled 
and analyzed at a meeting of the research team in December, 1988. 
For the purposes of cross-site analysis, data froa the individual 
sites were "standardized" (Miles & Huberman, 1984, p. 152) by 
means of common categories and common 7, sporting formats (i.e., 
summary forms, analysis fots^j, case reports) . Ajialysis o.-f che 
standardized data took place in four stages: (a) development of a 
neta-Jiatrix, (b) clustering of data, (c) identification of 
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patterns, and (d) d»>relQpiaent of propositions (Miles & Huberman, 
1984) . 

In developing meta-raatrices, "the basic principle is the 
basic inclusion of all relevant data" (Miles & Huberman, 1984, p. 
152) . For the purposes of this study, a meta-matrix was 
developed by the project director from summaries of within-site 
analyses. Data from each institution were described by 
categories: (a) the role of institutional agents regarding out- 
of-class expairiences, (b) description and role of student 
subcultures, (c) description and role of institutional history 
and traditions, (d) description and role of institutional 
policies and practices, (e) description and role of institutional 
mission, (f) characteristics of student involvement in out-of- 
class life, (g) tentative explanations and speculations. 
Institution-specific categories were subsumable into the original 
set of categories. 

Once the meta-matrix was prepared, the research team members 
proceeded to cluster data in order to identify commonalities and 
differences in categories across sites (Miles & Huberman, 1984) . 
In addition, commonalities and differences were described 
according to the five types of institutions: small residential, 
large residential, urban-commuter, single-sex, historically 
black. Thus, we were able to discover that, except in the case 
of urban-commuter institutions, the various institutional types 
had many more elements in common than differences. 

The cluster o^. "things in common" was then examined in order 
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to identify patterns or themes emerging in each categoicy. Prom 
those themes / a set of propositions was developed to describe and 
explain, however tentatively, factors and conditions associated 
with high quality out-of-class experiences for undergraduates. 
Those propositions, then, were discussed in debrief ings with 
respondents and evaluated during the second round of site visits. 

In mid-March, 1989, after half of the second round visits 
were completed, the team met to analyze data from those visits 
and reconsider the propositions identified in December. As a 
resvit of new information (both from second rovmd interviews and 
member ''■hecks with respondents), the origin? set of factors and 
conditions was expanded and altered. Foi example, in December we 
had identified egalitarianism as a condition of an environment in 
which high-quality out-of-class experiences are present. m 
March, we recognized that, at some of the institutions, strong 
and clear— and non-egalitarian— distinctions were made among 
roles and statuses— distinctions that were consistent with 
institutional mission and that also appeared to contribute to 
student involvement. Thus, we revised our original proposition 
to focus on interpersonal distinctions in a variety of forms, all 
of which were consistent with the mission of the institution in 
which they were found. 

At our final team meeting in June, we looked at the data 
from the rest of the second round visits and incorporated 
additional information and insights into our propositions; at 
that point, our task was mostly one of filling in and refining, 
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rather than adding to or deleting from, the factors and 
conditions. The bulk of our time at that meeting was spent in 
trying to identify the policy and practice implications of our 
findings. 

At this point, a few words about "group process" are 
necessary. The foregoing description can be interpreted to 
portray a systematic, methodical, even dry, process of group 
analysis. Not so. Our team discussions were intense, lively, 
and challenging, characterized as much by argtiments as by 
agreement, as much by taking side roads as sticking to our 
planned route. Four rules did, however, emerge from our 
discussions: (a) everyone was expected to contribute to the 
process of analysis, (b) decisions (e.g., conclusions to be drawn 
from the data, propositions) made during data analysis were made 
by consensus, (c) thinking aloud was encouraged, and (d) every 
attempt should be made to avoid taking things personally (e.g., 
if, despite the rules, someone's- "bad" idea was attacked). on 
occasion, feathers were ruffled, but problems were confronted and 
dealt with openly; periodic breaJis to get away from one another 
(to run, take naps, etc.) were helpful. 

Team members took turns playing the roles of gatekeeper, 
encourager, questioner, and recorder (all of these meetings were 
taped as well). Some people talked more than others, some 
preferred to take notes and summarize, some had a high need to 
stay on task while others needed to ramble (although even the 
most task-oriented among us came to recognize that taking tours 
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down side roads often led^s to fruitful new territories). The 
project director was charged with making sure that we 
accomplished what we needed to, although we always needed more 
time together than we had. Follow-up "meetings" took place by 
phone or BITNET after we returned to our home institutions. 

Establishing Trustworthiness 
The principles of qualitative research described earlier-- 
e.g., seeking understanding, allowing the research design to 
emerge, using a human instrument to study natural settings, 
primary reliance on interviews with respondents—create problems 
for the researcher who seeks to conform to traditional standards 
of scientific inquiry (Crowson, 1987) . For example, 
replicability (necessary to establish reliability) is probably 
impossible and even, by ths standards of some inquiry paradigms, 
undesirable (Lincoln & Guba, 1985). Also, oral data can be 
particularly misleading as respondents knowingly or unknowingly 
try to place their experiences, ideas, and institution in the 
best possible light (Van JMaanen, 1979b) . The complexities and 
contradictions of institutions of higher education further 
mitigate attempts to accurately portray "reality" to the 
satisfaction of both natives and outsiders (Crowson, 1987) . "The 
central difficulty lies in the need to maintain the flexibility, 
opportunism, idiosyncracy, and holism required to maximize the 
inductive potential of an inquiry, while providing the many 
checks that lead the skeptical reader toward a sense of 
confidence in the study's report" (Crowson, 1987, p. 40) . 
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With these difficulties in mind, we adopted the standard of 
trustworthiness (Crdwson, 1987; Lincoln & Guba, 1985) to answer 
the question, "how can an inquirer persuade his or her audiences 
(including self) that the findings of an inquiry are worth paying 
attention to, worth taking account of?" (Lincoln & Guba, 1985, p. 
290). Criteria for trustworthiness include credibility (i.e., 
the constructions arrived at are credible to the respondents), 
transferability (i.e., the study may be useful in another 
contesct), dependability (i.e., the reporting of results considers 
possible changes over time), and confirmability (i.e., the data 
can be confirmed by someone other than the inquirer) (Lincoln & 
Guba, 1985) . Means for meeting the criteria for trustworthiness, 
are described below. 
Credibility 

Three of the mechanisms cited by Lincoln and Guba (1985) for 
establishing credibility (triangulation, peer debriefing, and 
member checks (debrief ings with respondents)) were used in this 
study. Triangulation is a technique for judging the accuracy of 
data, and requires the use of multiple data sources and/or 
multiple methods of data collection. Multiple sources of data 
may include multiple "copies" of one kind of source (e.g., 
multiple respondents) and different sources of the same 
information. m this study, data were obtained from five 
different types of institutions of higher education. m 
addition, at every institution, respondents in nine general 
categories (i.e., students, presidents, chief academic officers. 
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chief student affairs officers, faculty, student affairs staff, 
institutional historians, alumni/ae, and trustees) were 
interviewed. All nine types of respondents at all five types of 
institutions were asked to provide information about the out-of- 
class experiences of undergraduates, the role of institutional 
agents and policies in those experiences, and the connection, if 
any, between out-of-class experiences and tlie academic mission of 
the institution. 

Debriefing of the inquirer by a peer is used: (a) to ensure 
that the inquirer is aware of her or his personal perspectives 
and perceptions—the researcher's "perceptual screen" (Van 
Maanen, 1979b, p. 548) —and the impact they have on the study; 
(b) to develop and test next steps to be taken; and (c) to test 
hypotheses which are emerging from the data. Debriefing sessions 
were particularly critical for this study as nine "human 
instruments" were involved. First, visits to Wichita state and 
Grinnell were conducted by two teams of investigators in mid- 
September. 7ai nine members of the research team then met by 
conference telephone call to discuss the first visits, and make 
adjustments to interview protocols and other data-gathering 
techniques . 

Throughout the study, team members at each site met at the 
end of each day of interviews to discuss findings, plan for 
additional questions and respondents, and discuss tentative (and 
temporary) conclusions. Follow-up telephone (and BITNET) 
conferences were held in order to further discuss data and 
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impressions. 

Debrief ings were also conducted at research team meetings, 
held four times during the course of the study. These 
debriofings were used to test .Ideas, obtain feedback on methods 
(e.g., interview techniques, use of categories), and discuss next 
steps . 

Member checks are, in effect, debriefing sessions with 
respondents for the purpose of testing the data, analytical 
categories, interpretations, and conclusions; in short, for 
judging the overall credibility of the findings of the study 
(Lincoln & Cuba, 1985). Debrief ings with respondents occurred 
throughout the study and were informal as well as formal. At the 
end of most interviews, the investigators reviewed with the 
respondents what they had heard the respondents say, seeking 
immediate feedback and clarification of the interview. Also, 
after the first round of site visits, we began to "recycle" data 
among the respondents at each institution. Respondents from each 
category received a case report about their institution developed 
by the site team. Conversations were held with respondents, 
either by phone or in person during the second site visit, to 
obtain reactions to the questions, comments, concerns, and 
experiences described by respondents in the first round. This 
process served to focus later interviews and reinforce the 
constructions that were emerging through data analysis. case 
reports developed from the second round of visits were also 
circulated among respondents for their reactions, which were then 
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incorporated in to final case reports. 

Finally, copies of our preliminary propositions regarding 
factors and conditions ass ciated with high quality out-of-class 
experiences of undergraduates were sent to respondents at all lA 
sites. Their reactions to the propositions were used in the 
process of developing conclusions for the stv.dy as well as to 
inform the second round of site visits - 
Transferability 

To address tlie issue of transferability (i.e., can the 
findings of the study be used in another context?), the inquirer 
must demonstrate the degree of similarity between the sending 
(i.e., the setting of the study) and receiving (i.e., the setting 
to which the study may be applied) contexts (Lincoln & Guba, 
1985). Therefore, she or he must provide a thick description of 
the sending context so that someone in a potential receiving 
context may assess the similarity between them and, hence, the 
transferability of the findings. Thick description entails the 
broadest and most thorough information possible (Lincoln & Gtiba, 
1985). In reporting the findings and conclusions of the study, 
we have provided as accurate descriptions of the settings and 
respondents as concern for confidentiality allowed, as well as an 
extensive discussion of factors and conditions and themes across 
institutions, including statements from which they were derived. 
Dependab llitv and Conflrmabllity 

In order to meet criteria for dependability (i..e., the 
reporting of results considers possible changes over time) , the 
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inquirer must provide evidence of the appropriateness of the 
inquiry decisions made throughout the study (Lincoln & Cuba, 
1985). Confirmability (i.e., the data can be confirmed by 
someone other than the inquirer) of the data is demonstrated by 
showing that the findings are based on the data and that the 
inferences drawn from the data are logical (Lincoln & Guba, 
1985). Dependability and confirmability can be established by 
means of an audit, in which an external auditor examines both the 
processes and the products of the study. During the course of 
the study, we have developed an audit trail (Lincoln & Cuba, 
1985) comprised of: (a) raw data, including tapes, interview 
notes, and documents; (b) products of data reduction and 
analysis, including field notes, interview and document summary 
forms, and case analysis forms; (c) products of data 
reconstruction and synthesis, including category descriptions, 
case reports, and ongoing reports of findings and conclusions; 
(d) process notes, including notes on methodological decisions 
and trustworthiness criteria; and (e) materials relating to the 
intention and disposition of the research team, including notes 
of debrief ings, staff meeting minutes and tapes, and staff 
correspondence . 

FINDINGS AND CONCLUSIONS 
Now that the study has been completed, four conclusions can 
be drawn about the research approach and methods we used. First, 
multisite qualitative research using multiple investigators is 
time-consuming. The collection of data for a rich description of 
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one site and information needed for institutional comparisons 
requires extended time on site by several investigators, doing 
interviews and observing the life of the campus. Some would say, 
perhaps, that our two rounds of t -ee- and four-day visits cannot 
be considered "extended" time on s«te. We would certainly agree 
that there is more we could have learned with more time and money 
(a topic we will discuss below), but we are satisfied that cur 
results are credible to participants in the setting and add to 
knowlr^dge about how involvement in out of class experiences 
promotes student learning and personal development. 

The process of getting thorough and complete (or as complete 
as possible) feedback from respondents at the site also demanded 
a lot of time. Sending reports to the respondents, waiting for 
them to have time to read the reports, and waiting for their 
written comments took weeks (and even, in some cases, months). 
Thus, patience and commitment to the necessity of this input are 
crucial . 

In addition, analysis of data across sites and the process 
of eliciting factors and conditions that contribute to high 
quality out-of-class experiences for students demanded four two- 
and three-day team meetings in which site reports and individual 
constructions were thoroughly discussed. The process of 
achieving some shared vision—that is, shared among ourselves as 
well as shared between investigators and "natives"— of the places 
we visited and their commonalities and differences was protracted 
and intense. 
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Our visits also involved significant time and effort on the 
part of the institutions we visited, our contact persons 
developed interview schedules, mailed sets of documents to site 
team members, arranged for meeting rooms and (in many cases) 
lodging, made sure that we were picked up from and delivered to 
airports, and helped us to cope with any crises that arose (e.g., 
cancelled interviews, dead tape recorder batteries). The 
hospitality and assistance of the people at the sites was a 
tremendous help to us and was a highlight of the study 
experience. 

Second, multisite research involving a team of investigators 
can be (and was) expensive in terms of money as well as time. 
The study was funded by three grants (from the Lilly Endowment, 
the National Association of Student Personnal Administrators, and 
the Education Services Division of the Marriott Corporation) , one 
of which was very substantial. without external funding for 
travel and lodging for nine team members; half-time secretarial 
assistance (for document management, project correspondence, tape 
transcriptions, and preparation of project reports) ; two half- 
time graduate assistants (for travel arrangements, library 
research); tapes, batteries, and ten leased tape recorders; and 
copying costs, we could not have done such a large-scale and in- 
depth study. What we could have done with less money, what we 
could have done without, or even what we could have done with 
more funds, are debateable questions. Researchers interested in 
doing this type of study should tailor their budgets to their 
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research purposes and work from there to seek funding. Something 
to keep in mind is the ever-changing cost of air travel and the 
tremendous variation in fares, depending on when and where you 
must fly. 

Third, the composition of the research team was absolutely 
critical to the success of the study. The study would not have 
been possible if all of the team members had not committed 
themselves to investing the time and effort needed to achieve our 
purposes as well as meet the demands of qualitative research; we 
all knew at the outset that we faced a complex and lengthy task, 
although just how complex and how lengthy emerged only as the 
study proceeded. Thus, another important qualification for 
involvement in this type of research (besides familiarity with 
the phenomena under study) is willingness to commit time not only 
to site visits, but to all of the processes of data reduction and 
analysis and writing and thinking and sharing and debating that 
follow. 

All of these tasks tested our stamina as well as our time 
schedules. Working with a focus group, doing an effective 
interview, and trying to understand a new setting require a lot 
of energy. However trite this may seem, anyone undertaking such 
a study should be conscientious about getting enough sleep and 
exercise and eating well. 

Some of the team members had more time to give to the 
project than others and so they were involved in more site visits 
and more of the follow-up writing tasks. Some of the team 
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members were less conscientious than others about completing 
summary forms and responding to requests for information. This 
will happen in any group project, and decisions need to be made 
in advance about how, or v helper, the group (or the project 
director) will respond. it should be noted, however, that all 
team members participated in the group meetings at which cross- 
site analysis occurred. 

More important, perhaps, than time, the team members 
(knowingly or unknowingly) committed themselves to learn (about 
qualitative research methods, about the sites, about themselves— 
their assumptions, values, attitudes—, about other coam 
members), to suspend judgment (about the institutions in the 
study, about the ideas and assumptions of other team members), 
and to share openly their ideas, questions, and feelings. For 10 
montiis, we worked together almost constantly, a situation that 
occasionally tried our patience and senses of humor, but also 
provided each of uj with a tremendously powerful and satisfying 
learning experience. 

Finally, despite the demands of this type of research, we 
believe that it works— that it has enabled us to obtain rich and 
accurate descriptions of the individual institutions and to 
identify factors and conditions across institutions that seem to 
contribute to high quality oiit-of-class experiences for 
undergraduate students. Evidence of our success, and the 
effectiveness of the method, can be founu in (a) feedback from 
students, faculty, and administrators at all of the sites that 
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our case reports are "very accurate" portrayals of their 
institutions, and (b) positive &nd affirming responses from 
various audiences (e.g., at conferences of the National 
Association of Student Pe^'-eonnel Administrators, American College 
Personnel Association, American Educational Research Association; 
site respondents) to the factors and conditions we have 
identified. 

An additional benefit of our reisearch has been the process 
of self-examination and self -discovery that our visits and 
reports have precipitated at the institutions we visited (e.g., 
"I hadn't really thought about this before."). our process of 
discovery and understanding has affected both the knower and the 
known. 
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